> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Anny26022/chartsmaze_clone/llms.txt
> Use this file to discover all available pages before exploring further.

# Phase 4: Enrichment Injection

> Modifies master JSON in-place to inject advanced metrics, F&O data, and event markers

## Overview

Phase 4 is the **final enrichment stage** where the base JSON from Phase 3 is modified in-place by 5 specialized scripts. Each script adds specific fields to complete the 86-field schema.

<Warning>
  **Order matters!** Scripts must run sequentially:

  1. `advanced_metrics_processor.py` (needs OHLCV)
  2. `process_earnings_performance.py` (needs filings + OHLCV)
  3. `enrich_fno_data.py` (needs external FNO data)
  4. `process_market_breadth.py` (needs returns + SMA status)
  5. `add_corporate_events.py` (MUST BE LAST — adds event markers + news)
</Warning>

## Execution Order

Phase 4 runs **5 scripts sequentially**:

<Steps>
  <Step title="Advanced Metrics Injection">
    `advanced_metrics_processor.py` — Injects ADR, RVOL, ATH, Turnover metrics
  </Step>

  <Step title="Earnings Performance Injection">
    `process_earnings_performance.py` — Injects post-earnings returns
  </Step>

  <Step title="F&O Data Injection">
    `enrich_fno_data.py` — Injects F\&O flag, lot size, next expiry
  </Step>

  <Step title="Market Breadth Processing">
    `process_market_breadth.py` — Generates sector analytics and breadth metrics
  </Step>

  <Step title="Corporate Events Injection (FINAL)">
    `add_corporate_events.py` — Injects event markers, announcements, news feed
  </Step>
</Steps>

***

## Script 1: advanced\_metrics\_processor.py

### Purpose

Injects OHLCV-derived metrics: ADR, RVOL, ATH, Turnover, Volume EMA.

### Input Files

<CodeGroup>
  ```plaintext Required theme={null}
  all_stocks_fundamental_analysis.json   (Phase 3 output)
  ohlcv_data/{SYMBOL}.csv                (Phase 2.5 output)
  complete_price_bands.json              (Phase 2 output)
  ```

  ```python Loading Logic theme={null}
  import json
  import pandas as pd
  import glob

  BASE_DIR = os.path.dirname(os.path.abspath(__file__))
  JSON_INPUT = os.path.join(BASE_DIR, "all_stocks_fundamental_analysis.json")
  OHLCV_DIR = os.path.join(BASE_DIR, "ohlcv_data")

  with open(JSON_INPUT, "r") as f:
      master_data = json.load(f)

  csv_files = glob.glob(os.path.join(OHLCV_DIR, "*.csv"))
  ```
</CodeGroup>

### Processing Logic

<CodeGroup>
  ```python Calculate ADR (Average Daily Range) theme={null}
  df = pd.read_csv(f"ohlcv_data/{symbol}.csv")

  # Daily range percentage
  df['Daily_Range_Pct'] = ((df['High'] - df['Low']) / df['Low']) * 100

  # Moving averages of ADR
  adr_5 = df['Daily_Range_Pct'].tail(5).mean()
  adr_14 = df['Daily_Range_Pct'].tail(14).mean()
  adr_20 = df['Daily_Range_Pct'].tail(20).mean()
  adr_30 = df['Daily_Range_Pct'].tail(30).mean()
  ```

  ```python Calculate RVOL (Relative Volume) theme={null}
  latest_volume = df['Volume'].iloc[-1]
  avg_vol_20 = df['Volume'].tail(21).iloc[:-1].mean()  # Exclude today

  rvol = latest_volume / avg_vol_20 if avg_vol_20 > 0 else 0
  ```

  ```python Calculate ATH (All-Time High) theme={null}
  ath = df['High'].max()
  latest_close = df['Close'].iloc[-1]

  pct_from_ath = ((ath - latest_close) / ath) * 100 if ath > 0 else 0
  ```

  ```python Calculate Turnover Metrics theme={null}
  df['Turnover_Cr'] = (df['Close'] * df['Volume']) / 10000000

  turnover_20 = df['Turnover_Cr'].tail(20).mean()
  turnover_50 = df['Turnover_Cr'].tail(50).mean()
  turnover_100 = df['Turnover_Cr'].tail(100).mean()
  avg_rupee_vol_30 = df['Turnover_Cr'].tail(30).mean()
  ```

  ```python Calculate Volume EMA theme={null}
  df['EMA_Vol_200'] = df['Volume'].ewm(span=200, adjust=False).mean()
  ema_vol_200_latest = df['EMA_Vol_200'].iloc[-1]

  # % from 52W high of 200D EMA Volume
  ema_vol_200_52w_high = df['EMA_Vol_200'].tail(252).max()
  pct_from_ema_200_52w_high = ((ema_vol_200_latest - ema_vol_200_52w_high) / ema_vol_200_52w_high) * 100
  ```
</CodeGroup>

### Fields Injected

| Field                                 | Description                        | Example    |
| ------------------------------------- | ---------------------------------- | ---------- |
| `RVOL`                                | Relative Volume (today vs 20D avg) | `1.45`     |
| `5 Days MA ADR(%)`                    | 5-day average daily range          | `3.2`      |
| `14 Days MA ADR(%)`                   | 14-day average daily range         | `3.5`      |
| `20 Days MA ADR(%)`                   | 20-day average daily range         | `3.4`      |
| `30 Days MA ADR(%)`                   | 30-day average daily range         | `3.6`      |
| `% from ATH`                          | Distance from all-time high        | `-12.5`    |
| `ATH_Value`                           | All-time high price                | `2850.00`  |
| `Gap Up %`                            | Today's gap vs yesterday close     | `1.2`      |
| `Day Range(%)`                        | Today's high-low spread            | `2.8`      |
| `6 Month Returns(%)`                  | 6-month price return               | `18.5`     |
| `% from 52W Low`                      | Distance from 52-week low          | `72.8`     |
| `30 Days Average Rupee Volume(Cr.)`   | 30-day avg turnover                | `1250.5`   |
| `Daily Rupee Turnover 20(Cr.)`        | 20-day avg turnover                | `1180.2`   |
| `Daily Rupee Turnover 50(Cr.)`        | 50-day avg turnover                | `1120.8`   |
| `Daily Rupee Turnover 100(Cr.)`       | 100-day avg turnover               | `1050.3`   |
| `200 Days EMA Volume`                 | 200-day EMA of volume              | `12500000` |
| `% from 52W High 200 Days EMA Volume` | Volume EMA trend                   | `-8.5`     |

### Threading

* **Workers:** 10 concurrent threads
* **Typical Time:** \~1-2 minutes (reading 2,775 CSV files)

### Dependency on OHLCV

<Warning>
  If `FETCH_OHLCV = False` in Phase 2.5, all these fields will remain **0**.
</Warning>

***

## Script 2: process\_earnings\_performance.py

### Purpose

Injects post-earnings price performance metrics.

### Input Files

<CodeGroup>
  ```plaintext Required theme={null}
  all_stocks_fundamental_analysis.json   (Phase 3 output, modified by Script 1)
  company_filings/{SYMBOL}_filings.json  (Phase 2 output)
  ohlcv_data/{SYMBOL}.csv                (Phase 2.5 output)
  ```

  ```python Detection Logic theme={null}
  import json
  import os
  import glob
  from datetime import datetime, timedelta
  import pandas as pd

  filings_dir = os.path.join(BASE_DIR, "company_filings")

  for filing in filings:
      caption = filing.get("caption", "").lower()
      if "quarterly" in caption and "results" in caption:
          # Found earnings filing
          filing_date = filing.get("news_date")
          break
  ```
</CodeGroup>

### Processing Logic

<CodeGroup>
  ```python Find Earnings Date from Filings theme={null}
  for filing in company_filings:
      caption = filing.get("caption", "").lower()
      if "quarterly" in caption and "results" in caption:
          earnings_date = datetime.strptime(filing["news_date"], "%Y-%m-%d")
          break
  ```

  ```python Calculate Returns Since Earnings theme={null}
  # Find pre-earnings close (day before filing)
  pre_earnings_close = df[df['Date'] < earnings_date].iloc[-1]['Close']
  current_price = df['Close'].iloc[-1]

  returns_since_earnings = ((current_price - pre_earnings_close) / pre_earnings_close) * 100
  ```

  ```python Calculate Max Returns Since Earnings theme={null}
  # Find peak price after earnings
  post_earnings_df = df[df['Date'] >= earnings_date]
  max_price = post_earnings_df['High'].max()

  max_returns_since_earnings = ((max_price - pre_earnings_close) / pre_earnings_close) * 100
  ```
</CodeGroup>

### Fields Injected

| Field                           | Description                                 | Example      |
| ------------------------------- | ------------------------------------------- | ------------ |
| `Quarterly Results Date`        | Date of latest earnings filing              | `2026-02-15` |
| `Returns since Earnings(%)`     | % change from pre-earnings close to current | `8.5`        |
| `Max Returns since Earnings(%)` | Peak % gain since earnings                  | `12.3`       |

### Typical Time

<Note>
  **\~2-3 minutes** — Reading 2,775 filing JSONs + CSV lookups
</Note>

***

## Script 3: enrich\_fno\_data.py

### Purpose

Injects F\&O (Futures & Options) metadata: lot size, next expiry, F\&O flag.

### Input Files

<CodeGroup>
  ```plaintext Required theme={null}
  all_stocks_fundamental_analysis.json   (Phase 3 output, modified by Scripts 1-2)
  fno_lot_sizes_cleaned.json             (External standalone script)
  fno_expiry_calendar.json               (External standalone script)
  ```

  ```python Loading Logic theme={null}
  import json
  from datetime import datetime

  with open("fno_lot_sizes_cleaned.json", "r") as f:
      lot_sizes = json.load(f)
      lot_map = {item["Symbol"]: item["Lot_Size"] for item in lot_sizes}

  with open("fno_expiry_calendar.json", "r") as f:
      expiry_data = json.load(f)
  ```
</CodeGroup>

### Processing Logic

<CodeGroup>
  ```python Inject F&O Flag theme={null}
  for stock in master_data:
      symbol = stock["Symbol"]
      
      # Check if symbol is in F&O list
      stock["Is FNO"] = 1 if symbol in lot_map else 0
  ```

  ```python Inject Lot Size theme={null}
  stock["FNO Lot Size"] = lot_map.get(symbol, "N/A")
  ```

  ```python Inject Next Expiry theme={null}
  today = datetime.now().date()
  upcoming_expiry = None

  for expiry in expiry_data:
      if expiry["Symbol"] == symbol:
          exp_date = datetime.strptime(expiry["Expiry_Date"], "%Y-%m-%d").date()
          if exp_date >= today:
              upcoming_expiry = expiry["Expiry_Date"]
              break

  stock["Next Expiry"] = upcoming_expiry if upcoming_expiry else "N/A"
  ```
</CodeGroup>

### Fields Injected

| Field          | Description                    | Example      |
| -------------- | ------------------------------ | ------------ |
| `Is FNO`       | 1 if F\&O enabled, 0 otherwise | `1`          |
| `FNO Lot Size` | Contract lot size              | `250`        |
| `Next Expiry`  | Next futures expiry date       | `2026-03-27` |

### Typical Time

<Note>
  **\~10-20 seconds** — Simple JSON lookups
</Note>

***

## Script 4: process\_market\_breadth.py

### Purpose

Generates sector-level analytics and relative strength ratings.

### Input Files

```plaintext theme={null}
all_stocks_fundamental_analysis.json   (Phase 3 output, modified by Scripts 1-3)
```

### Processing Logic

<CodeGroup>
  ```python Calculate Sector Breadth theme={null}
  sector_stats = {}

  for stock in master_data:
      sector = stock.get("Sector")
      if sector not in sector_stats:
          sector_stats[sector] = {
              "above_sma_50": 0,
              "above_sma_200": 0,
              "total_stocks": 0
          }
      
      sector_stats[sector]["total_stocks"] += 1
      
      if "Above" in stock.get("SMA Status", "") and "SMA 50" in stock.get("SMA Status", ""):
          sector_stats[sector]["above_sma_50"] += 1
      
      if "Above" in stock.get("SMA Status", "") and "SMA 200" in stock.get("SMA Status", ""):
          sector_stats[sector]["above_sma_200"] += 1
  ```

  ```python Calculate Relative Strength Rating (RSR) theme={null}
  # Rank stocks by 6-month returns
  sorted_stocks = sorted(master_data, key=lambda x: x.get("6 Month Returns(%)", 0), reverse=True)

  for idx, stock in enumerate(sorted_stocks):
      # RSR: 1-100 percentile rank
      rsr = int((1 - idx / len(sorted_stocks)) * 100)
      stock["Relative Strength Rating"] = rsr
  ```
</CodeGroup>

### Output Files

| File                    | Description                   |
| ----------------------- | ----------------------------- |
| `sector_analytics.json` | Sector-level breadth metrics  |
| `market_breadth.csv`    | Daily market breadth snapshot |

### Typical Time

<Note>
  **\~20-30 seconds** — In-memory calculations
</Note>

***

## Script 5: add\_corporate\_events.py (CRITICAL FINAL STEP)

### Purpose

**MUST BE LAST!** Injects event markers, regulatory announcements, and news feed.

### Input Files

<CodeGroup>
  ```plaintext Required theme={null}
  all_stocks_fundamental_analysis.json       (Phase 3 output, modified by Scripts 1-4)
  upcoming_corporate_actions.json            (Phase 2 output)
  company_filings/{SYMBOL}_filings.json      (Phase 2 output)
  market_news/{SYMBOL}_news.json             (Phase 2 output)
  nse_asm_list.json                          (Phase 2 output)
  nse_gsm_list.json                          (Phase 2 output)
  bulk_block_deals.json                      (Phase 2 output)
  incremental_price_bands.json               (Phase 2 output)
  ```

  ```python Loading Logic theme={null}
  import json
  import os
  import glob
  from datetime import datetime, timedelta

  BASE_DIR = os.path.dirname(os.path.abspath(__file__))
  master_file = os.path.join(BASE_DIR, "all_stocks_fundamental_analysis.json")
  upcoming_file = os.path.join(BASE_DIR, "upcoming_corporate_actions.json")
  filings_dir = os.path.join(BASE_DIR, "company_filings")
  asm_file = os.path.join(BASE_DIR, "nse_asm_list.json")
  deals_file = os.path.join(BASE_DIR, "bulk_block_deals.json")
  ```
</CodeGroup>

### Event Marker Logic

<CodeGroup>
  ```python 1. Surveillance Markers (★) theme={null}
  with open(asm_file, "r") as f:
      asm_data = json.load(f)

  for item in asm_data:
      symbol = item.get("Symbol")
      stage = item.get("Stage", "")
      
      if "LTASM" in stage:
          add_event(symbol, "★: LTASM")
      elif "STASM" in stage:
          add_event(symbol, "★: STASM")
  ```

  ```python 2. Corporate Action Markers (⏰, 💸, 🎁, ✂️, 📈) theme={null}
  with open(upcoming_file, "r") as f:
      upcoming_data = json.load(f)

  today = datetime.now()
  action_limit = today + timedelta(days=30)
  results_limit = today + timedelta(days=14)

  for event in upcoming_data:
      symbol = event.get("Symbol")
      etype = event.get("Type", "")
      edate_str = event.get("ExDate")
      
      edate = datetime.strptime(edate_str, "%Y-%m-%d")
      if today.date() <= edate.date() <= action_limit.date():
          d_str = edate.strftime("%d-%b")
          
          if "QUARTERLY" in etype and edate.date() <= results_limit.date():
              add_event(symbol, f"⏰: Results ({d_str})")
          elif "DIVIDEND" in etype:
              add_event(symbol, f"💸: Dividend ({d_str})")
          elif "BONUS" in etype:
              add_event(symbol, f"🎁: Bonus ({d_str})")
          elif "SPLIT" in etype:
              add_event(symbol, f"✂️: Split ({d_str})")
          elif "RIGHTS" in etype:
              add_event(symbol, f"📈: Rights ({d_str})")
  ```

  ```python 3. Filing-Based Markers (📊, 🔑) theme={null}
  for filing in filings:
      caption = filing.get("caption", "").lower()
      filing_date = datetime.strptime(filing.get("news_date"), "%Y-%m-%d")
      
      # Results marker (last 7 days)
      if "quarterly" in caption and "results" in caption:
          if (today - filing_date).days <= 7:
              add_event(symbol, "📊: Results Recently Out")
      
      # Insider trading marker (last 15 days)
      if "sebi reg 7(2)" in caption or "form c" in caption:
          if (today - filing_date).days <= 15:
              add_event(symbol, "🔑: Insider Trading")
  ```

  ```python 4. Block Deal Marker (📦) theme={null}
  with open(deals_file, "r") as f:
      deals_data = json.load(f)

  for deal in deals_data:
      symbol = deal.get("Symbol")
      deal_date = datetime.strptime(deal.get("Date"), "%Y-%m-%d")
      
      if (today - deal_date).days <= 7:
          add_event(symbol, "📦: Block Deal")
  ```

  ```python 5. Circuit Limit Revision (#) theme={null}
  with open(circuit_revision_file, "r") as f:
      rev_data = json.load(f)

  for item in rev_data:
      symbol = item.get("Symbol")
      from_band = float(item.get("From"))
      to_band = float(item.get("To"))
      
      if to_band < from_band:
          add_event(symbol, "#: -ve Circuit Limit Revision")
      elif to_band > from_band:
          add_event(symbol, "#: +ve Circuit Limit Revision")
  ```
</CodeGroup>

### Announcements Injection

```python theme={null}
# Top 5 regulatory filings
filings = load_filings(symbol)[:5]

announcements = []
for filing in filings:
    announcements.append({
        "Date": filing.get("news_date"),
        "Headline": filing.get("caption"),
        "URL": filing.get("pdf_url")
    })

stock["Recent Announcements"] = announcements
```

### News Feed Injection

```python theme={null}
# Top 5 news items with sentiment
news_items = load_news(symbol)[:5]

news_feed = []
for news in news_items:
    news_feed.append({
        "Title": news.get("title"),
        "Sentiment": news.get("sentiment"),  # positive/negative/neutral
        "Date": news.get("timestamp")
    })

stock["News Feed"] = news_feed
```

### Fields Injected

| Field                  | Description              | Example                                                                             |
| ---------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
| `Event Markers`        | Array of event strings   | `["★: LTASM", "💸: Dividend (15-Mar)", "📦: Block Deal"]`                           |
| `Recent Announcements` | Top 5 regulatory filings | `[{"Date": "2026-02-15", "Headline": "Quarterly Results", "URL": "..."}]`           |
| `News Feed`            | Top 5 news items         | `[{"Title": "Stock hits 52W high", "Sentiment": "positive", "Date": "2026-03-01"}]` |

### Event Marker Icons Reference

| Icon | Name                 | Trigger Condition                      |
| ---- | -------------------- | -------------------------------------- |
| ★    | Surveillance         | Stock in ASM/GSM lists                 |
| 📊   | Results Recently Out | Results filed in last 7 days           |
| 🔑   | Insider Trading      | SEBI Reg 7(2) / Form C in last 15 days |
| 📦   | Block Deal           | Bulk/Block deal in last 7 days         |
| #    | Circuit Revision     | Price band changed                     |
| ⏰    | Results Upcoming     | Results due in next 14 days            |
| 💸   | Dividend             | Dividend ex-date in next 30 days       |
| 🎁   | Bonus                | Bonus ex-date in next 30 days          |
| ✂️   | Split                | Split ex-date in next 30 days          |
| 📈   | Rights               | Rights issue in next 30 days           |

### Typical Time

<Note>
  **\~3-5 minutes** — Reading 2,775 filing JSONs + 2,775 news JSONs + event logic
</Note>

***

## Phase 4 Output Summary

### Final Master JSON

```plaintext theme={null}
📦 Phase 4 Final Output:
└─ all_stocks_fundamental_analysis.json   (~55 MB, 2,775 records, 86 COMPLETE fields)
```

### Field Completion Status

<CodeGroup>
  ```plaintext Before Phase 4 (Phase 3 Output) theme={null}
  ✅ 60 fields populated (fundamentals, technicals, ratios)
  ❌ 26 fields placeholder (0 or empty arrays)
  ```

  ```plaintext After Phase 4 (Final) theme={null}
  ✅ 86 fields FULLY populated
  ```
</CodeGroup>

***

## Total Phase 4 Execution Time

<Note>
  **\~6-10 minutes** (sum of all 5 scripts)
</Note>

### Breakdown:

* Script 1 (ADR/RVOL/ATH): \~2 min
* Script 2 (Earnings): \~3 min
* Script 3 (F\&O): \~20 sec
* Script 4 (Breadth): \~30 sec
* Script 5 (Events): \~4 min

***

## Critical Sequencing

<Warning>
  **Why order matters:**

  1. **advanced\_metrics\_processor.py first** — Needs raw OHLCV files
  2. **process\_earnings\_performance.py second** — Needs filings + OHLCV
  3. **enrich\_fno\_data.py third** — Independent lookup
  4. **process\_market\_breadth.py fourth** — Needs returns + SMA status from previous scripts
  5. **add\_corporate\_events.py LAST** — Adds final UI elements (markers, news)

  Running out of order will cause missing data or overwrites.
</Warning>

***

## Error Handling

Phase 4 uses **soft failure** mode:

```python theme={null}
results["advanced_metrics_processor.py"] = run_script("advanced_metrics_processor.py", "Phase 4")
# Pipeline continues even if enrichment fails
```

### Impact of Failures

* **Script 1 fails:** ADR, RVOL, ATH remain 0
* **Script 2 fails:** Earnings performance fields remain null
* **Script 3 fails:** F\&O fields remain N/A
* **Script 4 fails:** No sector analytics, no RSR
* **Script 5 fails:** Event markers, announcements, news feed remain empty

***

## Next Phase

Once Phase 4 completes, the pipeline proceeds to:

<Card title="Pipeline Architecture" icon="sitemap" href="/concepts/pipeline-architecture">
  See complete pipeline overview including Phase 5 compression details
</Card>

***

## Validation Checklist

After Phase 4, verify:

```bash theme={null}
# 1. Check file size (should be ~55 MB)
ls -lh all_stocks_fundamental_analysis.json

# 2. Validate field count
jq '.[0] | keys | length' all_stocks_fundamental_analysis.json  # Expected: 86

# 3. Check sample stock has all fields populated
jq '.[0]' all_stocks_fundamental_analysis.json | grep -E '(RVOL|Event Markers|Recent Announcements)'

# 4. Count stocks with event markers
jq '[.[] | select(."Event Markers" | length > 0)] | length' all_stocks_fundamental_analysis.json
```